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MARCH  2, 1981 


The  Honorable  Mark  O.  Hatfield 
Chairman,  Committee  on  Appropriations 
United  States  Senate 

The  Honorable  John  G.  Tower 
Chairman,  Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  William  V.  Roth,  Jr. 

Chairman,  Committee  on  Governmental 
Affairs 

United  States  Senate 

*  The  Honorable  Jamie  L.  Whitten 

Chairman,  Committee  on  Appropriations 
C S'\  House  of  Representatives 

The  Honorable  Melvin  Price 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  Jack  Brooks 
Chairman,  Committee  on  Government 
Operations 

House  of  Representatives 

Subject:  Recommendations  to  Improve  Defense 

*  Reporting  on  Weapon  Systems  (MASAD-81-7) 

We  issued  three  reports  1/  which  contained  recommendations 
to  the  Secretary  of  Defense  to  improve  the  Department  of 


lV"How  to  Improve  the  Selected  Acquisition  Reporting  System" 
(PSAD-75-63,  Mar.  27,  1975)  to  the  Congress;  " ' SARs ' — Defense 
Department  Reports  That  Should  Provide  More  Information 
to  the  Congress"  (PSAD-80-37,  May  9,  1980)  to  the  Congress; 
and  "DOD  Information  Provided  to  the  Congress  on  Major  Weapon 
Systems  Could  Be  More  Complete  and  Useful"  ( C-PSAD-80-24 , 

May  9,  1980)  to  the  chairman,  House  Committee  on  Government 
Operations . 
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Defense's  (DOD's)  reporting  to  the  Congress.  These  reports 
addressed  (1)  Selected  Acquisition  Reports  ( SARs ) ;  (2)  research, 
development,  test,  and  evaluation  (RDT&E)  descriptive  summaries’, 
(3)  Congressional  Data  Sheets?  (4)  other  budget  justification 
data,  and  (5)  testimony. 

Section  236  of  the  Legislative  Reorganization  Act  of  1970 
requires  an  agency  to  respond  to  our  reports  which  contain 
recommendations  and  state  what  actions  the  agency  plans  to  take 
to  implement  the  recommendations.  DOD  responded  to  our  May  1980 
reports  on  July  1,  1980,  and  September  18,  1980.  The  responses 
agreed  with  some  of  our  recommendations  and  identified  improve¬ 
ments  which  have  been  made  in  the  reporting  systems.  DOD,  how¬ 
ever,  disagreed  with  several  other  recommendations.  DOD  takes 
the  view  that  it  is  fully  complying  with  the  desires  of  the 
Congress  and,  therefore,  does  not  intend  to  implement  the  other 
recommendations.  Since  we  are  convinced  that  the  recommenda¬ 
tions  would  result  in  the  Congress  receiving  information  that 
would  be  valuable  in  carrying  out  its  oversight  responsibili¬ 
ties,  we  are  recommending  to  the  Congress  that  it  require  the 
Secretary  of  Defense  to  change  the  SAR  system  to 

— include  important  systems  that  are  in  advanced 
development;  1/ 

— expand  the  required  mission  capability  assessment 
statement  to  describe  shortcomings  and  limitations 
of  systems  in  their  expected  operational  environ¬ 
ments;  1/ 

— include  planning  estimates  and  a  one-time  variance 
analysis  for  the  planning  and  development  estimates 
in  the  first  report  that  includes  the  development 
estimate;  1/ 

— include  a  brief  narrative  section  on  technical  and 
operational  risks; 

— include  cost  estimates  for  categories  of  logistic 
support/additional  procurement  costs  related  to  the 
weapon  system  such  as  modification  costs,  component 
improvement  costs,  replenishment  spare  costs,  indus¬ 
trial  facilities/production  base,  simulators,  consum¬ 
ables,  and  modification  spares;  1/  and 


l/We  also  made  these  recommendations  to  the  Secretary  of 
Defense  in  our  March  1975  report. 
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— include  a  chart  showing  the  impact  on  the  program 
cost  estimate  of  different  escalation  rates. 


In  addition,  we  recommend  that  the  Congress  require  the 
Secretary  of  Defense  to  include  a  section  on  technical  and 
operational  risks  in  RDT&E  descriptive  summaries. 

We  are  convinced  that  implementing  these  recommendations 
would  result  in  improved  congressional  oversight  of  major 
weapon  systems  and  overall  better  management.  Much  of  the 
information  already  exists  and  could  be  incorporated  with 
little  additional  effort.  We  believe  that  any  additional 
costs  associated  with  providing  the  information  would  be 
justified. 

The  enclosure  to  this  letter  contains  DOD’s  comments 
on  our  original  recommendations  and  our  response  to  those 
comments.  Since  this  report  is  essentially  a  reiteration 
of  our  prior  reports,  we  did  not  ask  for  comments  from  DOD. 


We  are  sending  copies  of  this  report  to  the  Director, 
Office  of  Management  and  Budget,  and  to  the  Secretary  of 
Defense. 


Comptroller  General 
of  the  United  States 
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OUR  RESPONSE  TO  DEPARTMENT  OF  DEFENSE 

COMMENTS  ON  OUR  RECOMMENDATIONS 

EARLIER  SAR  REPORTING  OF 
IMPORTANT  MAJOR  SYSTEMS 


Our  recommendation:  The  Secretary  of  Defense  should  direct  that 
greater  consideration  be  given  to  (1)  adding  important  systems 
in  advanced  development  to  the  reporting  system  and  (2)  delet¬ 
ing  older  systems  from  the  reporting  system. 

POD  comment:  "The  March  1975  GAO  report  on  the  SAR  reporting 
system  recommended  that  the  DOD  establish  precise  criteria  for 
adding  major  acquisitions  to,  and  deleting  major  acquisitions 
from,  the  SAR  reporting  system.  We  agreed,  and  pursuant  to 
that  recommendation  the  SAR  instruction  was  revised  to  provide 
that  (1)  addition  of  a  new  SAR  would  be  automatic  with  DSARC 
[Defense  Systems  Acquisition  Review  Council]  II  approval  for 
the  system  to  enter  full-scale  engineering  development,  and 
(2)  termination  of  SAR  reporting  would  normally  be  considered 
when  production  of  the  system  is  90%  complete  and  the  program 
is  no  longer  a  procurement  budget  line  item.  We  believe  that 
these  criteria  were  very  precise  and  that  exceptions,  calling 
for  earlier  commencement  of  SAR  reporting,  should  be  at  the 
request  of  the  oversight  committees.  Termination  of  SAR 
reporting  is  now  coordinated  with  the  oversight  committees." 

Our  response:  The  important  issue  here  is  not  (as  implied  by 
the  DOD  comment)  whether  DOD  has  "precise"  criteria  for  adding 
and  deleting  systems.  The  issue  is  whether  the  Congress  and 
top  DOD  management  would  find  SAR  information  useful  on  major 
systems  which  have  not  yet  reached  DSARC  II.  Since  the  majority 
of  the  eventual  life  cycle  costs  of  a  weapon  system  are  deter¬ 
mined  by  decisions  made  before  DSARC  II,  we  believe  earlier  SAR 
reporting  is  necessary.  By  the  time  DSARC  II  rolls  around,  a 
system  is  often  so  "locked  in"  that  redirection  of  the  program 
is  practically  impossible.  Thus,  we  believe  all  top  level 
decisionmakers  need  this  vital  information  in  the  formative 
years  of  a  weapon  system's  development.  It  is  at  this  time 
that  judgments  are  made  which  affect  the  course  of  multibillion 
dollar  programs  for  years  to  come . 

MORE  MEANINGFUL  MISSION  ASSESSMENT  STATEMENT 


Our  recommendation:  The  Secretary  of  Defense  should  revise 
or  enforce  the  SAR  instruction  so  that  SARs  include  a  mission 
capability  assessment  statement,  including  expected  shortcom¬ 
ings  and  limitations  of  the  system  in  its  operational  environ¬ 
ment  . 
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TOD  comment :  "As  a  result  of  the  March  1975  GAO  report  on  the 
SAR  reporting  system,  we  revised  the  SAR  instruction  to  re¬ 
quire  a  statement  as  to  the  extent  the  system  is  expected  to 
satisfy  its  current  mission  requirement  and  to  identify  those 
areas  where  it  will  fall  short.  The  failure  to  include  this 
statement  in  12  of  the  51  SARs  is  an  implementation  problem 
and  will  be  rectified. 

"In  reference  to  including  additional  performance  characteris¬ 
tics,  we  believe  that  SAR  performance  characteristics  are 
specifically  tailored  to  the  system's  mission  requirements 
and,  thereby,  provide  the  basis  for  assessing  the  system's 
expected  effectiveness  in  attaining  its  mission  objectives. 
These  are  the  guidelines  provided  to  the  Project  Manager  at 
the  time  his  program  is  initially  designated  for  SAR  report¬ 
ing,  and  the  same  guidelines  are  used  by  Service  and  OSD 
[Office  of  the  Secretary  of  Defense]  review  authorities  in 
approving  the  performance  characteristics  developed  and  sub¬ 
mitted  by  the  Project  Manager.  Normally,  as  a  minimum,  the 
final  selection  includes  characteristics  for  which  DCP  [Deci¬ 
sion  Coordinating  Paper]  thresholds  exist,  the  principal  per¬ 
formance  requirements  of  the  weapon  system  contract,  and  other 
significant  characteristics.  In  several  instances,  we  have 
added  or  deleted  to  the  performance  characteristics  as  the 
mission  requirements  change.  In  view  of  the  above,  we  believe 
that  current  SAR  performance  characteristics  generally  conform 
with  the  requirements  of  the  GAO  recommendation.  if  the  over¬ 
sight  committees  develop  some  quantifiable  and  appropriate 
performance  characteristics  for  systems  which  they  feel  are 
deficient,  we  can  add  them  to  the  SAR.” 

Our  response:  TOD's  comment  implies  that  our  recommendation 
was  for  the  addition  of  performance  characteristics  in  SARs. 
This  was  not  our  intent.  Our  intent  was  that  the  mission 
capability  assessment  statement  be  expanded  to  include  a  clear 
statement  of  shortcomings  and  limitations  of  the  system  in 
its  expected  operational  environment.  Even  in  those  instances 
where— as  stated  by  DOD — the  "SAR  performance  characteristics 
are  specifically  tailored  to  the  system's  mission  requirements 
*  *  *"  we  believe  most  readers  of  SAR  would  be  unable  to  re¬ 
late  the  values  cited  to  how  well  the  system  is  expected  to 
perform  operationally. 

REPORTING  THE  PLANNING  ESTIMATE 


Our  recommendation;  The  Secretary  of  Defense  should  revise 
or  enforce  the  SAR  instruction  so  that  SARs  include  the 
planning  estimates  with  a  one-time  explanation  for  changes 
to  arrive  at  the  development  estimates. 
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POD  comment:  "Our  position  on  including  the  planning  estimate 
in  the  SAR  remains  unchanged  since  the  March  1975  GAO  SAR 
report.  The  GAO  report  misinterpreted  the  directive  on  this 
point.  It  covers  the  situation  when  establishing  a  SAR  prior 
to  approval  to  enter  full-scale  engineering  development 
( DSARC  II).  This  provision  was  necessary  because  in  the  past 
the  oversight  committees  have  requested  SAR  reporting  while  a 
system  was  in  advance  development.  Those  familiar  with  SAR 
reporting,  including  some  in  GAO,  understand  that  reporting 
of  the  planning  estimate  and  a  one-time  variance  explanation 
when  converting  to  a  development  estimate  is  only  necessary 
for  those  programs  whose  SARs  commenced  prior  to  the  DSARC  II 
approval.  Planning  estimates  are  created  prior  to  the  defini¬ 
tion  of  the  system  configuration  (which  makes  pricing  a  system 
with  reasonable  confidence  levels  difficult)  and  prior  to  the 
Secretary  of  Defense  decision  to  pursue  fully  the  program  as 
an  acquisition  program.  At  this  point,  the  DOD  has  not  yet 
closed  out  any  of  its  options  for  acquiring  a  specific  system, 
and  planning  estimates  reflect  cost  performance  and  schedule 
characteristics  of  a  "generic"  system.  Since  this  is  an 
exploratory  phase,  program  specifications,  costs,  and  quanti¬ 
ties  must  be  preliminary  and  will  be  more  fully  developed 
before  selecting  one  or  more  alternatives  for  full-scale 
development.  This  is  in  consonance  with  OMB  [office  of  Man¬ 
agement  and  Budget]  Circular  A-109,  which  states  that  care 
should  be  exercised  during  the  initial  steps  of  the  acquisi¬ 
tion  process  not  to  conform  mission  needs  or  program  objec¬ 
tives  to  any  known  systems  or  products  that  might  foreclose 
considerations  of  alternatives." 

Our  response:  We  disagree  with  the  DOD  comment.  We  fully 
understand  the  current  practice  within  DOD  for  reporting  the 
planning  estimates  and  described  this  practice  in  our  report. 
We  disagree,  however,  that  our  report  misinterpreted  the 
instruction  on  this  point.  Section  C.6.  of  the  SAR  instruc¬ 
tion  states  that 

"*  *  *  the  first  report  containing  the  DE  [devel¬ 
opment  estimate]  will  include  both  the  PE  [plan¬ 
ning  estimate]  and  the  DE  and  provide  a  one-time 
variance  analysis  of  the  differences  between  the 
PE  and  the  DE . " 

The  instruction  does  not  state  that  this  requirement  applies 
only  when  PE  has  been  reported  on  previous  SARs.  Regardless 
of  this  disagreement,  however,  we  made  the  above  recommenda¬ 
tion  because  we  believe  that  in  the  interest  of  fully  docu¬ 
menting  major  systems,  the  planning  estimate  (in  whatever 
form  it  exists)  should  be  reported  in  SAR  and  a  one-time 
variance  analysis  should  be  included  in  the  first  SAR  showing 
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DE  for  all  programs.  We  have  maintained  for  some  time  that 
the  Congress  is  entitled  to  know,  and  should  be  advised,  of 
what  the  planning  estimate  was.  We  view  the  planning  estimate 
as  the  baseline  for  measuring  future  progress.  It  is  also 
often  the  estimate  that  is  provided  to  the  Congress  at  the 
time  initial  funding  is  requested.  We  recognize  that  early 
estimates  are  often  difficult  to  make  with  precision  and 
have,  as  a  matter  of  fact,  suggested  that  such  estimates  be 
presented  in  the  form  of  ranges.  DOD  rejected  that  proposal 
stating  that  critics  tend  to  abuse  the  range  estimate,  using 
the  high  or  low  side  of  the  estimate  when  making  points  to 
detract  from  the  system. 

SAR  REPORTING  OF  OPERATIONAL  AND 
TECHNICAL  RISKS 


Our  recommendation:  The  Secretary  of  Defense  should  revise 
the  SAR  instruction  to  require  that  SARs  include  a  section 
on  operational  and  technical  risks. 

DOD  comment:  "We  appreciate  the  need  for  Congress  to  be  aware 
of  the  operational  and  technical  risks  of  weapon  systems.  We 
feel  that  this  information  should  be  provided  at  the  time  the 
Congress  reviews  and  approves  funding  for  the  system.  Since 
SARs  are  not  required  until  approval  of  full-scale  engineering 
development,  the  reporting  of  the  operational  and  technical 
risks  will  not  be  on  a  timely  basis.  We  feel  that  in  the 
event  there  is  a  request  for  this  information,  a  proper  time 
and  procedure  could  be  established  to  provide  the  required 
data.  " 

Our  response:  While  DOD's  comment  agrees  that  the  Congress 
needs  these  data,  it  questioned  whether  SAR  is  the  proper 
place  for  presenting  it  sine'-  ?ARs  are  normally  not  prepared 
until  after  DSARC  II.  Our  r*  idation  for  earlier  (pre- 

DSARC  II)  SAR  reporting  would  overcome  this  objection.  We 
feel  our  concern  would  be  satisfied,  however,  if  the  RDT&E 
descriptive  summary  included  a  narrative  on  operational  and 
technical  risks  such  as  is  now  included  in  DOD's  internal 
decision  coordinating  papers  for  many  major  systems. 

SAR  REPORTING  OF  LOGISTIC  SUPPORT/ 

ADDITIONAL  PROCUREMENT  COSTS 


Our  recommendation:  The  Secretary  of  Defense  should 
revise  the  SAR  instruction  to  require  reporting  of  logistic 
support/additional  procurement  costs  and  explanations  for 
changes . 
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POD  comment:  "This  recommendation  was  made  in  the  March  1975 
GAO  report  and  our  position  remains  unchanged.  These  types 
of  costs  are  functions  of  ownership  rather  than  acquisition 
and  are  outside  of  the  direct  control  of  the  Project  Managers. 
Often  logistic  support  and  replenishment  spare  costs  are 
related  to  changing  resource  availability  and  operational 
considerations.  To  include  those  costs  in  the  SAR  baseline 
would  make  variance  analysis  and  tracking  meaningless, 
particularly  when  they  are  not  directly  related  to  acquisi¬ 
tion  and  when  most  of  the  costs  accrue  after  SAR  reporting 
is  terminated." 

Our  response :  Logistic  support/additional  procurement  costs, 
as  described  in  our  report,  would  be  reported  as  a  separate 
category  of  costs  and  therefore  would  not  impact  on  the  SAR 
baseline  or  variance  analysis.  Because  tradeoffs  to  keep 
acquisition  costs  down  often  result  in  higher  operating  costs, 
we  believe  the  broader  view  provided  by  including  these  costs 
in  SAR  would  make  SARs  more  useful  to  the  Congress  in  its 
dec i s ionmaking . 

REPORTING  COSTS  BASED  ON  SEVERAL 
DIFFERENT  ESCALATION  RATES 


Our  recommendation:  The  Secretary  of  Defense  should  revise 
the  SAR  instruction  to  require  a  chart  showing  the  impact  on 
the  program  cost  estimate  of  using  different  escalation  rates. 

POD  comment:  "As  the  GAO  report  indicates,  this  information 
was  provided  in  SARs  prior  to  June  30,  1979.  The  detailed 
review  and  coordination  which  led  to  the  latest  SAR  revision 
revealed  no  need  or  request  for  this  information.  The  pro¬ 
posed  revision  to  the  instruction  was  provided  to  the  Congres¬ 
sional  oversight  committees  before  the  final  decision  was 
made  to  remove  the  escalation  chart.  Since  this  was  a  time- 
consuming  chart  to  prepare,  and  difficult  to  understand  we 
decided  to  replace  it  with  information  that  was  more  relevant. 
Therefore,  it  was  replaced  with  tracking  of  the  Design-to-Cost 
goal .  " 

Our  response:  The  DOD  comment  implies  no  congressional  objec¬ 
tions  to  the  revised  instruction.  However,  the  absence  of 
congressional  objections  to  the  revised  instruction  cannot, 
in  our  opinion,  be  interpreted  to  mean  that  the  information 
provided  by  the  cited  chart  was  not  useful.  Current  cost 
estimate  increases  for  the  F/A-18,  as  an  example,  clearly 
demonstrates  the  importance  of  more  realistic  consideration 
of  inflation.  We  strongly  believe  cost  estimates  showing  a 
more  relevant  range  of  escalation  projections  would  not  only 
be  useful  to  the  Congress  but  would  add  more  credibility  to 
DOD's  reporting. 
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REPORTING  OPERATIONAL  AND  TECHNICAL 
RISKS  IN  RDT&E  DESCRIPTIVE  SUMMARIES 

Our  recommendation:  The  Secretary  of  Defense  should  include 
an  operational  and  technical  risk  section  in  RDT&E  descriptive 
summaries  for  major  systems,  including  descriptive  summaries 
for  major  systems  not  yet  in  full-scale  engineering  development. 

POD  comment :  "The  definition  of  risk  is  to  a  large  extent 
open  to  individual  interpretation.  Because  of  this,  caution 
should  be  taken  in  attempting  to  specifically  categorize 
the  risk  of  a  program  in  subjective  terms,  i.e.,  low,  medium, 
or  high.  However,  risk  is  a  consideration  in  the  development 
of  test  and  evaluation  plans.  The  degree  or  magnitude  of 
the  testing  required  is  a  measure  of  risk. 

"The  descriptive  summaries  for  major  systems  contain  a  test 
and  evaluation  section  which  describes  the  degree  of  testing 
required  to  insure  that  a  program  meets  technical  performance 
specifications.  This  section  also  may  contain  comments  on  the 
degree  of  risk  if  it  can  be  determined  for  the  overall  system. 

"It  is  therefore  not  necessary  to  add  an  additional  section 
to  the  descriptive  summary  to  specifically  discuss  risk.  This 
would  add  a  measure  of  duplication,  and  unnecessarily  increase 
the  size  of  the  descriptive  summary." 

Our  response:  We  disagree  with  DOD's  comment.  We  believe 
RDT&E  descriptive  summaries  do  not  present  a  clear  and  concise 
statement  of  technical  and  operational  risks.  As  stated 
in  our  response  to  the  similar  SAR  recommendation  above, 
internal  DOD  documents  make  a  much  clearer  presentation  of 
risk.  Such  a  section  would  not,  therefore,  be  a  duplication 
of  data  already  in  the  descriptive  summaries,  especially  for 
systems  that  have  completed  little  or  no  developmental  or 
operational  testing. 
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